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FOREWORD 


The  Training  and  Simulation  Technical  Area  (Performance  Readiness  Team) 
of  the  U.  S.  Army  Research  Institute  has  been  actively  supporting  research 
designed  to  investigate  the  transfer  effectiveness  and  cost  effectiveness  of 
Computer  Based  Instruction  (CBI)  in  the  Army  school  environment.  A  key  ques¬ 
tion  has  been  the  degree  to  which  CBI,  through  the  use  of  simulations  and 
similar  programs,  can  substitute  for  training  on  actual  equipment,  thereby 
reducing  training  equipment  costs  and  increasing  the  number  of  students  who 
can  be  concurrently  trained.  This  report  describes  a  developmental  project 
conducted  at  the  Army  Signal  School,  Fort  Gordon,  Georgia.  A  computer  simu¬ 
lation  program  was  developed  to  teach  troubleshooting  skills  on  the  SB3614 
switchboard  to  MOS  36H  trainees  (Dial/Manual  Central  Office  Repairer)  as  a 
substitute  for  practice  on  the  equipment  itself.  The  results  from  this 
training  experience  thus  add  to  the  data  base  needed  to  determine  the 
utility  of  CBI. 


EDGAR  M.  JOHNSON 
Technical  Director 
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IMPLEMENTATION  OF  A  MIXED-FIDELITY  APPROACH  TO  MAINTENANCE  TRAINING 


EXECUTIVE  SUMMARY 


Requirement : 

To  document  the  conceptualization,  development,  integration,  and  pilot 
test  results  of  a  computer  based  simulation  for  troubleshooting  training  in 
an  Army  electronics  environment.  The  work  specifically  studied  the  value  of 
providing  a  variety  of  training  techniques  with  mixed  levels  of  fidelity. 


Procedure : 

The  research  contained  the  major  components  of  systematic  design  of 
courseware  including  front-end  analysis,  definition  of  diagnostic  training 
goals,  identification  of  appropriate  instructional  techniques,  development 
with  subject  matter  experts,  and  formative  and  summative  evaluation.  This 
work  resulted  in  a  demonstration  and  pilot  evaluation  of  a  microprocessor- 
based  troubleshooting  simulation. 


Findings : 

The  project  demonstrated  that  a  computer-based  simulation  with  both  low 
and  moderate  levels  of  fidelity  can  be  an  acceptable  supplement  to  high- 
fidelity  real  equipment  training  in  an  Army  electronics  environment.  The 
pilot  test  showed  the  methodology  and  feasibility  of  evaluating  computer- 
based  simulation  training.  The  results  of  the  pilot  test  suggest  that  the 
computer-based  simulation  can  be  exchanged  for  a  portion  of  the  real  equip¬ 
ment  training . 


Utilization  of  Findings: 

The  findings  provide  the  framework  for  similar  development  and  integra¬ 
tion  projects.  The  pilot  evaluation  results  and  methodology  provide  guidance 
for  a  significant  and  extensive  evaluation. 
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The  menu  of  choices  a  student  has  with  the  simulation  is 
shown  in  Figure  1 .  Items  numbered  three  through  seven  in  the 
figure  make  up  the  " symptom  finding"  section  of  the  simulation. 
These  items  contain  the  medium-fidelity  portions  of  the 
simulation  along  with  some  low- fidelity  portions.  Section  nine 
is  the  low-fidelity  section  in  which  failed  parts  may  be 
replaced.  Finally,  section  eight  contains  the  Framework  for 
Aiding  the  Understanding  of  Logical  Troubleshooting  (FAULT). 


Hon  to  use  the  simulation. 


Information  for  form  1105  and 

TELEPHONE  TYPES. 


Check  and  change  the  terminal 

CONNECTIONS  AND  SNITCH  POSITIONS. 


Check  and  change  PC  card  installation 
AND  SNITCH  POSITION  FOR  EACH  SUBSCRIBER. 


Observe  the  front  panel  and  perform 
LAMP  TEST. 


Will  the  snitchboard  accept  and  verify 
THE  PROGRAM? 


Perform  operator,  subscriber  and 
TRUNK  CALLS. 


Use  logic-based  charts  and  the  computer 
to  find  the  failure. 


Replace  the  failed  part. 


Figure  1.  Main  menu  list  of  options. 


The  process  of  gathering  symptom  information  via  the 
simulation  is  necessarily  different  than  on  real  equipment.  The 
simulation  does  not  attempt  to  teach  such  high-fidelity  skills  as 
turning  knobs  or  flipping  switches.  It  is  assumed  that  students 
in  the  SB  3614  class  already  possess  these  skills  or  will  acquire 
them  in  working  with  the  real  equipment. 
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and  Duncan's  Multiple  Range  tests.  In  the  analysis,  experts 
a  within-sub j ects  variable  and  training  groups  was 

between-subj ects  variable. 


a 


The  analysis  of  variance  showed  a  significant  interaction 
between  the  experts  and  groups  for  both  the  composite,  F  (3,37)  = 
24,  £  <  0.05,  and  the  action-by-action,  F  (3,37)  =  36,  p  <  0.05, 
rating.  For  the  composite  rating,  Duncan's  Multiple  Range  test 
indicated  that  two  of  the  experts  rated  the  experimental  group  as 
having  better  overall  performance,  £  <  0.05*  The  other  two 
experts'  ratings  showed  no  statistical  difference  between  the 
groups.  For  the  action-by-action  rating,  Duncan's  Multiple  Range 
test  indicated  that  one  expert  rated  the  experimental  group  as 
having  significantly  better  performance  than  the  control  group,  p 
<  0.05.  The  other  three  experts'  action-by-action  ratings  showed 
no  difference  between  the  groups. 


There  was  a  significant  main  effect  of  experts  for  the 
action-by-action  rating,  £  <  0.05*  This  difference  among  experts 
may  be  attributed  to  each  expert’s  differing  understanding  of  the 
instructions  for  completing  the  rating  forms  or  to  the  different 
standards  of  acceptable  performance  of  each  expert.  Subsequent 
use  of  this  rating  scheme  must  attempt  to  eliminate  this 
difference  by  increased  expert  education  with  respect  to  the 
rating  scheme  or  by  scaling  the  experts'  ratings. 


3.1.3  Fine-grained  Error  Analysis 

The  experts'  ratings  were  studied  more  closely  in  an  effort 
to  determine  the  specific  types  of  error  committed  during  the 
live  performance  test.  An  error  was  judged  to  be  any  student 
action  which  an  expert  rated  as  "inappropriate"  or  "most 
inappropriate".  It  was  found  that  there  were  83  actions  which  at 
least  one  expert  considered  to  be  in  error.  Two  or  more  experts 
agreed  that  errors  had  been  committed  for  29  of  the  actions.  It 
was  decided  to  consider  only  those  actions  for  which  at  least  two 
experts  agreed  that  errors  had  been  committed  since  the  analysis 
in  the  previous  section  showed  that  there  were  differences 
between  the  experts. 


The  system  used  to  classify  the  various  errors  was  that  used 
bv  Johnson  and  Rouse  (1982).  With  this  system,  errors  may  be 
classified  into  five  general  categories.  These  categories  are 
defined  as  follows. 

1.  Observation  of  State:  Occurred  when  a  student  failed  to 
collect  sufficient  information,  misinterpreted  the 
information  collected,  or  collected  the  same  information 
more  than  once  before  forming  initial  hypotheses  about 
the  cause  of  the  symptoms. 

2.  Choice  of  Hypotheses:  Occurred  when  a  student  chose  a 
hypothesis  that  may  have  been  functionally  related  to 
tne  symptoms  but  was  a  poor  choice  because  of  the  nature 
of  the  specific  symptoms,  a  very  low  probability  of 
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3.  PILOT  EVALUATION 


The  primary  objectives  of  the  pilot  evaluation  were  to  test 
the  training  effectiveness  of  the  SB  3614  simulation  and  to 
determine  the  extent  to  which  the  simulation  is  understandable  to 
the  user.  This  section  summarizes  the  results  of  the  evaluation. 


3 . 1  Live  Performance  Test 

The  live  performance  test  analysis  was  the  most  important  in 
determining  the  training  effectiveness  of  the  SB  3614  simulation. 
The  global  measures  (solution  success  and  time  to  solution), 
expert  ratings,  and  fine-grained  errors  analyses  revealed 
progressively  more  detailed  information  concerning  student 
performance  in  both  the  experimental  and  control  groups.  Results 
from  each  of  these  analyses  are  described  below. 

3.1.1  Global  Measures 


Ratings  of  solution  success  and  time  to  solution  for  the 
live  system  performance  test  are  global  measures  in  the  sense 
that  they  provide  a  performance  overview.  For  the  test,  the  mean 
rates  of  solution  failure  were  9^>  and  7%  for  the  control  and 
experimental  groups  respectively.  Due  to  the  small  difference 
between  these  rates,  no  further  analysis  was  performed. 

Time  to  solution  comparisons  were  based  on  all  133  solutions 
to  the  three  live  performance  problems.  Mean  solution  times  for 
both  groups  are  shown  in  Table  1 .  A  t-test  was  performed  for 
each  problem  to  determine  whether  there  was  a  difference  between 
the  two  groups.  For  one  of  the  three  problems,  average  solution 
time  for  the  experimental  group  was  significantly  less  (£  <  0.10) 
than  that  of  the  control  group. 


Table  1 

Mean  Solution  Times  (in  minutes)  for  Live  Performance  Test 


Problem 

Control 

Experimental 

Significance 

1 

14-0 

11 .6 

to 

/v 

o 

• 

o 

2 

11.7 

11.4 

3 

10.3 

9-3 

3.1.2  Expert  Ratings 

Analysis  of  the  experts'  ratings  of  student  performance  on 
the  live  system  performance  test  included  an  analysis  of  variance 
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Performance  measures  for  the  live  equipment  test  were 
solution  success  and  time  to  solution  for  each  problem. 
Experience  with  evaluation  of  live  system  troubleshooting  has 
indicated  that  such  global  measures  are  not,  in  themselves,  the 
best  measures  from  which  to  make  comparisons  between  groups 
(Johnson  and  Rouse,  1982a).  Therefore,  a  troubleshooting 
performance  checklist  was  used  to  record  the  step-by-step  actions 
of  39  students  (20  control  and  19  experimental)  solving  one 
particular  problem.  These  checklists  were  analyzed  by  four  SB 
3614  experts  who  rated  each  individual  student  action  and  then 
assigned  a  composite  rating  to  the  solution  as  a  whole.  A  total 
of  T03  control  group  actions  and  113  experimental  group  actions 
were  rated  in  this  manner.  An  overall  performance  rating  was 
then  determined  using  the  action-by-action  ratings  in  a  manner 
similar  to  that  described  by  Johnson  (1980). 

After  completion  of  the  live  equipment  troubleshooting  test, 
the  experimental  group  was  given  an  opinionnaire .  This 
opinionnaire  questioned  the  students  regarding  previous  computer 
experience,  their  feelings  concerning  the  simulation  and  the 
generalizability  of  their  SB  3614  training  to  other  equipment. 
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2.3  Experimental  Design  and  Procedure 


As  stated  previously,  two  classes  of  subjects  were  used  in 
the  pilot  study.  Non-experimental  SB  3614  classes  receive 
approximately  12  hours  of  real  equipment  troubleshooting 
practice.  Both  control  and  experimental  classes  received 
approximately  five  additional  hours  of  training,  for  a  total  of 
17  hours  of  training.  Previous  experience  with  training 
evaluations  (Johnson,  1980)  suggested  that  five  additional  hours 
of  troubleshooting  training  and/ or  practice  should  have  a 
resultant  effect  on  performance.  For  the  control  group  this  five 
hours  consisted  of  three  hours  of  real  equipment  troubleshooting 

Sractice  and  two  hours  of  work  on  a  paper  and  pencil  exercise. 

xperimental  group  students  were  given  a  1  1/2  hour  introduction 
to  the  SB  3614  simulation.  This  was  followed  by  approximately  3 
1/2  hours  of  troubleshooting  practice  on  the  simulation. 


The  program  of  instruction  was  not  changed  to  accomodate  the 
experiment,  since  a  requisite  of  this  study  was  to  minimize 
disruption  of  normal  instruction.  Further,  instructors  were  the 
same  for  each  class  so  that  instructor  differences  would  not  bias 
the  evaluation  results. 


2.3.1  Simulation  Training 

Following  the  midterm  examination,  students  in  the 
experimental  group  completed  six  sessions  of  computer  practice  in 
a  three  day  training  period.  The  computer-based  simulation 
training  was  intermingled  with  the  use  of  real  equipment.  This 
arrangement  permitted  the  student  to  mix  practice  on 
high-fidelity  real  equipment  with  practice  on  the  mixed-fidelity 
simulation.  Each  computer  session  lasted  from  27  to  40  minutes. 
During  the  experiment,  students  solved  nearly  700  problems  on  the 
simulation  for  a  total  of  90  hours  of  problem  solving  time.  On  a 
per  student  basis,  this  corresponds  to  an  average  of  23  problems 
solved  in  three  hours  (Approximate  time  at  the  learning  station 
was  3.5  hours).  The  mean  solution  time  was  7.8  minutes  with  a 
range  of  one  to  24  minutes. 

2.3.2  Performance  Measures 

Simulation  performance  data  collected  included  the  problem 
number,  time  to  solution,  and  a  rough  accounting  of  the  errors 
made  in  working  each  problem.  Diskette  space,  and  other 
logistical  problems  related  to  the  Apple,  minimized  the  amount  of 
data  collected  for  each  problem. 

Post-training  data  for  both  groups  focused  on  performance  on 
a  live  equipment  troubleshooting  test.  In  the  test,  each  student 
was  to  correctly  identify  two  out  of  three  live  system  failures 
in  no  more  than  20  minutes  per  problem.  During  the  test,  63 
students  solved  133  live  system  problems  in  a  12  hour  period. 
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include  the  ability  to  make  logical  troubleshooting  tests  and  to 
view  a  system  as  a  group  of  interconnected  parts.  The 
low-fidelity  FAULT  section,  then,  teaches  students  general 
cognitive  troubleshooting  skills. 


t|  6 


Figure  3.  Digital  tone  generation  subsystem. 


2.2.3  Documentation 


Two  user  documents  are  provided  for  the  SB  3614  simulation. 
These  are  the  Student  Manual  for  SB  3614  and  the  Instructor 
Manual  for  SB  3614. 

The  purpose  of  the  Student  Manual  is  not  only  to  introduce 
the  student  to  the  simulation,  but  also  to  the  Apple  II  computer. 
All  necessary  equipment  for  the  simulation  is  listed  in  this 
manual.  Further,  students  are  instructed  as  to  the  starting 
procedure  for  the  simulation.  Following  these  preliminaries,  the 
manual  explains  how  and  where  in  the  simulation  to  find  problem 
symptoms.  It  then  describes  FAULT  in  detail.  Also  included  in 
the  manual  is  a  section  containing  forms  to  be  completed  if  a 
student  finds  an  error  in  the  simulation. 

The  Instructor  Manual  serves  several  purposes.  The  most 
important  function  is  to  provide  instructors  with  materials  for 
introduction  of  the  simulation  to  the  students.  For  this 
purpose,  the  Instructor  Manual  includes  guidelines  for 
integration  of  the  simulation  into  the  program  of  instruction,  a 
list  of  points  to  be  emphasized  during  an  introduction  of  the 
simulation,  and  display  facsimiles  to  be  used  as  overhead  slides. 
Another  purpose  of  the  manual  iB  to  provide  instructors  with 
failure  information.  This  information  includes  a  list  of  the 
possible  failures,  an  operator's  symptom  report  for  each  failure, 
and  the  failure  sequence  for  each  lesson  set.  The  final  purpose 
of  the  Instructor  Manual  is  to  provide  instructors  with 
information  concerning  the  use  of  the  instructor  program. 
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Figure  2.  Real  and  simulated  switchboard  rear  panel. 


2.2.2  FAULT 

The  eighth  item  in  the  menu  of  choices  in  Figure  1  is  System 
Troubleshooting  —  FAULT.  FAULT  is  a  direct  product  of  the 
research  base  described  in  Section  1.4*  In  the  FAULT  section, 
students  are  allowed  to  view  SB  3614  subsystems  on  a  conceptual 
level  using  one  of  seven  FAULT  subsystem  diagrams.  With  the  help 
of  these  diagrams,  the  student  may  make  troubleshooting  tests  for 
a  subsystem  in  which  a  failure  is  suspected. 

Figure  3  shows  a  FAULT  diagram  for  the  Digital  Tone 
Generation  Subsystem.  The  diagram  is  a  low-fidelity 
representation,  since  each  subsystem  component  is  depicted  by  a 
named  and  numbered  box.  Relationships  between  parts  are 
represented  by  lines  between  the  boxes. 

In  the  FAULT  section,  students  are  allowed  to  make 
observations  between  parts,  bench  test  parts,  and  replace  parts. 
In  addition,  students  may  access  nearly  oO  part  descriptions,  as 
well  as  information  on  the  easiest  way  to  test  each  component. 
Further,  FAULT  aids  students  in  troubleshooting  by  informing  them 
of  errors  in  logic.  When  such  an  error  is  made,  (i.e.,  testing  a 
known  good  part)  FAULT  notifies  the  student  of  the  error.  For 
■ore  information  concerning  FAULT,  see  Hunt  and  Rouse  (1981). 

In  working  a  problem  with  FAULT,  a  student  learns 

Seneralizable  skills  as  well  as  specific  SB  3614  information. 

pacific  information  learned  includes  testing  procedures  and  part 
names,  relationships,  and  functions.  Generalizable  skills 
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The  simulation  stresses  the  enhancement  of  cognitive 
troubleshooting  ability  over  development  of  manual  skills.  For 
instance,  the  first  step  a  student  might  take  in  gathering 
symptom  information  is  to  check  the  terminal  box  on  the  back  of 
the  switchboard.  To  do  this  on  real  equipment,  the  student 
merely  looks  at  the  terminal  box  on  the  back  of  the  switchboard. 
With  the  simulation,  the  student  must  make  a  menu  selection  to 
see  the  terminal  box  representation.  Further,  for  real 
equipment,  checking  the  terminal  box  connections  involves 
confirming  that  all  connections  are  pushed  securely  into  place  as 
well  as  checking  that  the  connections  are  correct  as  per  the 
specific  telephone  system  connection  chart.  With  the  simulation, 
however,  it  is  assumed  that  the  connections  are  already  pushed 
tightly  into  place.  Therefore,  the  student  has  only  to  establish 
that  the  connections  are  correctly  positioned. 

Another  example  of  the  difference  between  high-fidelity  real 
equipment  and  medium-fidelity  simulation  is  demonstrated  in 
Figure  2.  The  left  half  of  Figure  2  shows  the  real  equipment 
switchboard  rear  panel,  while  the  right  half  shows  the  simulated 
version.  In  the  simulated  version  the  switches  do  not  move,  and 
the  knob  does  not  turn,  however  it  contains  all  of  the  important 
information  displayed  on  the  real  equipment.  In  a 
troubleshooting  situation,  students  merely  note  the  positions  of 
the  controls,  thus,  for  the  rear  panel,  all  important  information 
is  conveyed  through  a  medium-fidelity  representation. 

Once  the  symptoms  are  found  and  the  failure  is  identified, 
the  student  must  replace  the  failed  part.  Parts  may  be  replaced 
in  the  simulation  section  entitled  "Parts  Replacement".  In  this 
section,  a  list  of  parts  is  presented  and  the  student  is  required 
to  choose  the  number  of  the  failed  part.  In  addition,  the 
student  must  identify  the  subsystem  which  includes  the  failed 
part . 


The  fidelity  of  this  section  is  very  low,  since  only  the 
part  names,  not  their  images,  are  shown.  Further,  typing  the 
number  of  a  part  to  be  replaced  is  nothing  like  physically 
removing  the  part  from  the  SB  3614  system  and  replacing  it  with  a 
good  part.  The  simulation  stresses  only  the  cognitive  skill  of 
identifying  the  failed  component,  since  physical  part  replacement 
is  a  straightforward  task. 
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Eleven  t-tests  were  performed  to  determine  the  comparability 
of  the  two  classes  based  on  the  two  measures  described  above. 
Midterm  grades  were  found  to  have  no  statistical  difference.  One 
out  of  the  ten  ASVAB  composite  scores  was  significant  (p  <  0.05). 
Based  on  the  fact  that  ten  out  of  the  11  t-tests  indicated  no 
significant  difference  between  the  two  classes,  it  was  assumed 
that  the  classes  were  from  the  same  statistical  population. 


2.2  Materials 

Materials  developed  for  the  SB  3614  course  included  both  a 
mixed-fidelity  computer-based  simulation  and  the  associated 
documentation.  The  simulation  has  two  main  sections;  a 
"symptom-finding"  section  in  which  students  can  gather 
information  indicative  of  failures  in  the  SB  3614  and  a  section 
containing  FAULT  in  which  students  may  troubleshoot  subsystems  of 
the  SB  3614.  The  simulation,  written  in  Pascal,  uses  a  64K  Apple 
II  Plus,  two  disk  drives,  a  clock,  and  a  monochrome  monitor. 
Documentation  for  the  simulation  includes  both  instructor  and 
student  manuals. 

2.2.1  The  Simulation 

The  mixed-fidelity  approach  lends  itself  well  to  simulation 
of  the  SB  3614  for  several  reasons.  First  of  all, 
medium-fidelity  portions  of  the  simulation  provide  a  visual  link 
between  the  simulation  and  real  equipment.  Students  readily 
recognize  graphical  representations  of  SB  3614  components  and  can 
relate  them  to  the  real  equipment.  Secondly,  low-fidelity 
representations  are  sufficient  to  present  certain  types  of 
information.  For  example,  information  concerning  the  state  of  a 
light  (i.e.,  "on"  or  "off")  can  often  be  described  in  words 
rather  than  pictured  through  graphics.  Finally,  a  mix  of 
fidelity  allows  the  embedding  of  higher-level  problem  solving 
concepts  in  the  simulation.  These  embedded  concepts  can  then  be 
generalized  to  troubleshooting  of  other  equipment. 

This  section  describes  the  SB  3614  simulation  with  an 
emphasis  on  the  two  levels  of  fidelity.  For  a  more  detailed 
description  of  the  simulation  and  its  operation,  consult  the 
Student  Manual  for  SB  3614  Computer  Simulation  and  Instructor's 
Manual  for  SB  3614  Computer  Simulation. 

The  purpose  of  the  computer-based  SB  3614  simulation  is  to 
allow  students  to  practice  troubleshooting  SB  3614  failures.  In 
troubleshooting  simulated  SB  3614  problems,  as  in  solving  real  SB 
3614  problems,  the  student  must  first  gather  information 
concerning  the  symptoms  of  the  current  failure.  Having  this 
information,  the  student  can  proceed  to  check  components  in 
suspect  subsystems  until  the  problem  is  solved. 
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2.  METHOD 


The  mixed-fidelity  concept  was  demonstrated  via  a 
computer-based  simulation  of  the  SB  3614*  Using  this  simulation, 
an  Initial  evaluation  was  performed  with  two  groups  of  students 
enrolled  in  the  SB  3614  course.  This  section  presents 
information  concerning  the  students  who  participated  in  the 
evaluation,  the  courseware  developed,  the  experimental  design  for 
the  initial  evaluation,  and  the  procedure  for  the  evaluation. 


2.1  Sub.1  ects 

Most  students  trained  on  the  SB  3614  at  the  Signal  School  in 
Fort  Gordon,  Ga.  are  in  the  Army  36H  MOS,  the  Dial/Manual 
Central  Office  Repairer.  Soldiers  with  this  MOS  have 
responsibilities  related  to  the  installation  and  maintenance  of 
military  telephone  exchanges.  In  order  to  be  accepted  into  the 
36H  MOS  students  are  usually  high  school  graduates  with  above 
average  intelligence  and  reasonably  high  Armed  Services 
Vocational  Aptitude  Battery  (ASVAB)  scores. 

Through  informal  student  interviews,  the  authors  observed 
that  36H  students  typically  choose  this  MOS  in  order  to  receive 
valuable  electronics  training  and  on-the-job  experience  that  will 
provide  saleable  skills  in  the  civilian  sector.  Thus,  students 
are  quite  motivated  and  anxious  to  develop  competency  related  to 
installation,  operation,  and  repair  of  the  equipment.  This 
motivation  is  especially  high  during  the  SB  3614  training,  which 
presents  students  with  their  first  opportunity  during  the  course 
to  use  totally  solid  state  equipment.  Unlike  much  of  the  older 
electro-mechanical  equipment  used  throughout  the  training,  the  SB 
3614  equipment  represents  new  technology  to  the  students. 

The  pilot  evaluation  was  performed  with  two  groups  of 
students.  The  control  group  contained  35  students  and  the 
experimental  group  contained  30.  The  average  age  of  students  in 
both  groups  was  22  years.  Average  civilian  education  of  the 
students  was  12  1/2  years. 

Two  measures  were  used  to  determine  whether  the  two  groups 
were  matched  prior  to  the  experiment.  These  measures  were  the 
ten  ASVAB  composite  scores  and  class  midterm  examination  grades. 
Composite  scores  are  combinations  of  various  ASVAB  subtest 
scores.  Such  data  were  available  for  all  Army  students  in  the 
two  classes.  For  more  information  concerning  ASVAB  scores,  refer 
to  Maier  and  Grafton  (1981). 

The  midterm  is  a  written  test  designed  to  measure  the 
student's  overall  knowledge  of  switchboard  operation,  basic 
construction,  and  various  subsystem  functions.  It  was 
administered  after  the  first  six  days  of  SB  3614  instruction. 
Midterm  grades  were  available  for  all  students  in  the  two 
classes. 


designed  for  portability  and  for  use  under  rugged  conditions. 
The  electronics  of  the  SB  3614  are  solid  state  and  represent 
relatively  state-of-the-art  technology.  The  cost  of  each  SB  361 4 
is  approximately  $40,000.  The  equipment  will  remain  as  a 
standard  for  small  systems  communications  for  quite  some  time. 
Though  it  is  used  by  all  the  branches  of  the  military,  the  Army 
and  Marine  Corps  employ  the  most  systems. 


aircraft  simulators  create  a  highly  realistic  flying  experience 
through  use  of  motion  and  of  cockpit  visual  displays.  If 
lower-fidelity  simulation  could  provide  some  of  the  same  aspects 
of  training  as  does  high-fidelity  simulation,  then  aircraft 
simulators  could  be  much  less  costly.  The  same  holds  true  for 
any  type  of  simulation. 

Full-scope,  high-fidelity  simulators  are  potentially 
effective,  in  terms  of  training  and  coBt,  for  equipment  such  as 
aircraft  or  nuclear  power  plants.  In  these  applications,  the 
high  cost,  unavailability,  and  danger  associated  with  real 
equipment  makes  simulation  training  an  •  absolute  necessity. 
However,  many  of  the  knowledge  and  skill  requirements  nurtured 
during  training  are  not  necessarily  best  accomplished  through 
high-fidelity  simulation.  For  example,  the  pilot  trainee  does 
not  need  a  high-fidelity  simulator  to  learn  how  to  use  the 
aircraft's  navigational  equipment. 

Research  related  to  mixed-fidelity  training  should  strive  to 
develop  a  method  to  determine  the  optimal  mix  of  training  to 
capitalize  on  traditional  instructional  techniques  and  the  use  of 
a  variety  of  simulation  training  methods.  ThiB  project  attempts 
to  apply  the  mixed-fidelity  concept  to  electronics  training.  The 
results  may  be  generalized  to  a  variety  of  technical  training 
environments . 


1 . 5  Selecting  Demonstration  Hardware 

In  selecting  a  hardware  system  for  demonstration  of  the 
mixed-fidelity  concept,  there  were  a  number  of  considerations. 
Ideally,  the  hardware  choice  would  result  in  the  development  of 
needed  and  useful  training  courseware  while  demonstrating  and 
evaluating  the  mixed-fidelity  concept.  Therefore  the  objective 
was  twofold,  to  satisfy  the  research-oriented  goals  while 
fulfilling  operational  training  requirements. 

Consideration  of  both  research  and  operations  missions  led 
to  the  choice  of  the  SB  3614  tactical  switchboard  as 
demonstration  hardware.  The  SB  3614  equipment  is  relatively  new, 
being  introduced  into  the  Army  communications  hardware  inventory 
in  1976.  Therefore,  the  training  developments,  if  proven 
valuable,  had  the  potential  for  extensive  use  and  would  serve  to 
justify  the  development  costs.  Additionally,  the  annual  number 
of  students  trained  on  the  equipment  in  the  13  day  course  is 

fuite  high,  ranging  between  250  and  300  at  Fort  Gordon  alone. 

his  high  student  throughput  was  especially  important  for 
evaluation  of  the  courseware. 

The  technical  manual  (TM11-5805-695-12)  describes  the  SB 
3614  (V)/TT  as  a  tactical,  ruggedized,  30-terminal  automatic 
switchboard.  The  SB  3614  weighs  approximately  65  pounds  and  has 
dimensions  of  15  inches  by  36  inches  by  24  inches.  It  is 
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A  series  of  ten  experiments  involved  over  300  subjects 
solving  over  24,000  fault  diagnosis  problems.  The  details  of  the 
experiments  are  reported  elsewhere  (Rouse  and  Hunt,  1984).  The 
first  five  experiments  addressed  context-free  problem  solving. 
The  next  three  considered  the  relationships  between  context-free 
and  context-specific  problem  solving.  The  final  two  experiments 
studied  the  transfer  of  training  from  the  computer  simulations  to 
actual  live  equipment  in  the  domain  of  aircraft  powerplants. 

A  general  conclusion  from  the  experiments  was  that  training 
for  problem  solving  can  be  enhanced  with  computer-based 
simulation.  There  was  a  positive  transfer  from  generalized 
context-free  simulation  to  context-specific  simulation  and  from 
context-specific  simulation  to  live  aircraft  equipment.  Further, 
variations  in  the  types  of  computer  aiding  and  the  types  of 
context-specific  information  provided  during  training  had  an 
effect  on  the  type  of  performance  errors  made  during  live 
equipment  troubleshooting  (Johnson  and  Rouse,  1982,  a  and  b) . 

One  of  the  most  important  outcomes  of  the  early  research  at 
the  University  of  Illinois  was  the  concept  of  mixed-fidelity 
training.  Briefly,  mixed-fidelity  training  involves  the 
identification  of  a  variety  of  instructional  techniques  to 
accomplish  the  various  types  of  knowledge  and  skill-related 
objectives  necessary  for  diagnostic-related  training.  This 
concept  will  be  discussed  in  Section  1.4. 

The  early  work  was  quite  rigorous  in  terms  of  empirical 
evaluation  and  involved  numerous  subjects  and  use  of  real 
equipment.  However,  it  did  not  take  place  in  an  operational 
military  training  environment.  Questions  left  unanswered 
addressed  the  extent  to  which  the  mixed-fidelity  training 
approach  could  be  successfully  conceptualized,  developed, 
implemented  and  evaluated  at  an  operational  Army  school.  This 
project,  therefore,  sought  to  answer  these  questions. 


1 .4  Mixed-Fidelity  Training 

While  the  phrase  "Mixed-Fidelity  Training"  (Rouse,  1982)  may 
be  somewhat  unique,  the  notion  is  by  no  means  a  revolution  in 
technical  training.  The  most  important  attribute  of 
mixed-fidelity  training  is  the  conscious  effort  on  the  part  of 
the  instructional  designer  to  match  the  fidelity  of  the  training 
devices  to  particular  Instructional  objectives. 

Fidelity  is  the  extent  to  which  a  training  device  has  the 
appearance  and  behavior  of  real  equipment.  The  topic  of  fidelity 
has  been  thoroughly  addressed  by  Hays  (1981).  The  majority  of 
research  related  to  simulation  fidelity  has  concerned 
high-fidelity  aircraft  simulators.  Because  of  the  potentially 
substantial  cost  of  high-fidelity  simulators  in  this  domain, 
level  of  fidelity  is  a  critical  issue.  For  example,  some 
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has  the  potential  to  alleviate  the  problem.  Finally,  the 
training  must  be  designed  and  delivered  such  that  there  is  a 
reasonable  expectation  of  learning  retention. 


1 . 2  Proi ect  Goals 

This  project  had  several  goals,  which  are  listed  below: 

1 .  Build  upon  previous  Army-funded  research  to  move  the 
work  from  the  University  laboratory  into  an  operational 
Army  training  environment. 

2.  Work  with  Army  electronics  training  personnel  to 

conceptualize,  design,  and  implement  a 

simulation-oriented  computer-based  device  for  diagnostic 
training. 

3.  Design  an  equipment  specific  training  device  that  also 
delivers  a  degree  of  generic  diagnostic  instruction. 

4.  Develop  a  computer  simulation  that  includes  more  than 
one  level  of  fidelity . 

5.  Conduct  a  pilot  evaluation  of  the  simulation's 
diagnostic  training  effectiveness. 


The  next  section  describes  the  evolving  research  effort  that 
led  to  this  project  and  offers  a  rationale  for  the  above  goals. 
In  addition,  the  following  sections  more  clearly  define  some  of 
the  terms  introduced  above. 


1 .3  Diagnostic  Training  Research  -  Rationale  for  Continuation 

Research  concerning  the  design,  implementation,  and 
evaluation  of  diagnostic-related  training  programs  has  a  lengthy 
history  and  has  been  repeatedly  discussed  and  reviewed  (Johnson, 
1980;  Johnson,  Rouse,  and  Rouse,  1980;  Orlansky  and  String, 
1981;  Standlee,  et.  al.,  1956).  This  section  will  discuss  the 
work  of  the  authors  and  their  colleagues,  since  that  research 
established  the  empirical  foundation  of  this  project. 

The  roots  of  this  applied  research  at  the  Signal  School  of 
Fort  Gordon,  Georgia  began  with  ARI  sponsored  work  at  the 
University  of  Illinois  from  1978  through  1981  (Rouse  and  Hunt, 
1984).  The  research  ran  the  gamut  from  basic  to  specific  context 
application  for  aviation  mechanics'  training  (Johnson,  1980; 
Johnson  and  Rouse,  1982  a  and  b).  The  extensive  empirical 
findings  demonstrated  various  strengths  and  weaknesses  of  the 
computer-based  diagnostic-related  training  simulations  (Rouse  and 
Hunt,  1984). 


1.  BACKGROUND  FOR  IMPLEMENTATION  OF  MIXED-FIDELITY  TRAINING 


1 . 1  The  Diagnostic  Performance  Dilemma 

Equipment  maintenance  has  placed  a  heavy  burden  on  financial 
and  manpower  resources  in  all  branches  of  the  military  (Orlansky 
and  String,  1981).  It  is  generally  recognized  that  the  results 
of  ineffective  or  inadequate  troubleshooting  of  complex 
mechanical  and  electronic  equipment  contributes  significantly  to 
the  maintenance  costs.  Inefficiency  is  reflected  in  excessive 
manpower  allocation  to  diagnostic  maintenance  tasks  and  excessive 
erroneous  replacement  of  components  that  have  not  failed.  The 
part  replacement  problem  creates  the  demand  for  vast  spare  part 
inventories  and  clogs  the  repair  pipeline  with  non-failed 
components.  Of  even  greater  significance  is  the  fact  that 
deficient  diagnostic  performance  threatens  the  operational 
readiness  of  critical  communications  systems  and  other  equipment 
for  land,  sea,  and  air  uses. 

The  recent  school  graduate  takes  the  brunt  of  the  criticism 
regarding  on-the-job  diagnostic  inability.  The  standing 
complaint  from  the  field  has  been  that  recent  graduates  placed  in 
the  operational  environment  cannot  troubleshoot.  When  a 
maintenance  task  is  diagnostic  in  nature  it  is  usually  the 
experienced  technician  who  is  summoned.  The  problem  is 

►  confounded  by  the  in-service  course  offerings  related  to 

troubleshooting.  When  a  special  troubleshooting  course  is 
offered,  quite  often  a  prerequisite  to  course  attendance  is  a 
minimum  rank  or  minimum  years  of  experience.  This  situation 
i  creates  the  conflict  that  in  order  to  take  the  troubleshooting 

course  one  must  be  experienced  and  already  know  how  to 
„  troubleshoot. 

j 

j  Various  technical  schools  throughout  the  Army  have  the 

j  responsibility  of  providing  soldiers  with  entry-level  skills  and 

I  knowledge  for  the  assigned  military  occupational  specialty  (MOS) . 

Since  each  MOS  is  responsible  for  a  variety  of  equipment,  it  is 
!  quite  obvious  that  not  every  recent  graduate  is  likely  to  be  an 

"expert"  on  all  equipment  in  the  MOS.  It  is  not  atypical  for  a 
soldier  to  be  trained  on  specific  equipment  in,  for  example,  the 
fourteenth  week  of  a  40  week  course  and  then  not  see  the 
equipment  again  until  20  weeks  into  the  tour  of  duty.  In  this 
]  example  there  is  nearly  a  one  year  delay  between  training  and  the 

demand  for  on-the-job  performance. 

i 

!  Since  it  is  not  possible  to  prevent  the  delay  between 

training  and  field  performance,  other  action  must  be  taken  to 
i  minimize  potential  performance  deficiencies.  In  many  cases  the 

'  Job  must  be  structured  to  provide  recurrent  training  or  review 

and  practice  of  seldom  used  skills.  Another  solution  is  to 
provide  adequate  job  aids  to  prompt  refamiliarization  with 
equipment.  Embedding  training  into  operational  equipment  also 


being  true,  or  a  very  high  cost  of  testing.  Also 
occurred  when  a  hypothesis  was  functionally  unrelated  to 
the  symptoms. 

3.  Choice  of  Procedures:  Occurred  when  a  student's  choice 

of  procedure,  including  informal  procedures,  was 
incomplete  or  inappropriate  with  respect  to  the 

hypothesis  being  tested.  Also  occurred  when  a 
systematic  procedure  was  not  adopted. 

4.  Execution  of  Procedures:  Occurred  when  a  student 

omitted  procedural  steps,  performed  steps  out  of 
sequence,  etc.,  or  committed  apparently  inadvertent 
actions . 

5.  Consequence  of  Previous  Error:  Occurred  when  an  error 
was  a  logical  consequence  of  a  previous  error . 


The  29  errors  were  classified  using  the  five  error 
categories.  Results  of  the  error  classification  are  shown  in 
Table  2.  These  results  are  reported  because  they  are  suggestive 
of  the  types  of  results  which  will  be  pursued  in  the  formal 
evaluation.  They  should  not  be  regarded  as  definitive  results 
for  two  reasons.  First,  as  stated  in  the  previous  section,  there 
was  a  difference  in  the  expert  action-by-action  ratings. _  This 
difference  raises  questions  about  the  experts'  understandings  of 
the  rating  scheme.  The  second  reason  is  the  small  sample  size. 
Larger  student  samples  must  be  examined  before  definitive  results 
may  be  obtained. 

From  Table  2,  it  is  seen  that  18  out  of  29,  or  63%  of  the 
errors  were  committed  by  students  in  the  control  group.  This 
represents  17%  (18/103)  of  the  total  number  of  control  group 
troubleshooting  actions.  Ten  per  cent  (11/11 3)  of  the  actions 
taken  by  the  experimental  group  were  rated  as  errors.  In 
performing  a  statistical  comparison  of  these  proportions  (Glass 
and  Stanley,  1970)  it  was  found  that  this  difference  is 
significant  at  the  £  <  0.05  level.  Therefore,  in  comparing  the 
actions  of  the  experimental  and  control  groups,  it  can  be 
concluded  that  a  greater  percentage  of  the  actions  taken  by 
control  group  students  were  errors. 
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Table  2 

Fine-grained  Error  Analysis  Results 


General  Category 

Specific  Category 

Control 

Experiment 

1 . 

Observation  of 

a. 

incomplete 

1 

0 

System  State 

b. 

misinterpreted 

0 

0 

c . 

repeated 

3 

5 

2. 

Choice  of 

a . 

inconsistent 

3 

1 

Hypothesis 

b. 

unlikely 

0 

0 

c. 

costly 

0 

0 

d. 

irrelevant 

1 

0 

3. 

Choice  of 

a. 

incomplete 

0 

0 

Procedures 

b. 

inappropriate 

9 

5 

c. 

lack 

0 

0 

4. 

Execution  of 

a. 

omission 

0 

0 

Procedures 

b. 

other 

0 

0 

c. 

inadvertent 

0 

0 

5. 

Consequence  of  Previous 

Error 

1 

0 

The  two  types  of  errors  most  commonly  committed  were  the 
repeated  observation  of  state  (1c)  and  the  failure  to  choose  the 
appropriate  procedures  (3b).  More  repeated  observation  of  system 
state  errors  were  committed  in  the  experimental  than  the  control 

froup.  The  simulation  does  not  penalize  students  for  accessing 
he  same  section  more  than  once.  Thus,  the  simulation  may  not 
sufficiently  stress  thoroughness  in  cnecking  displays  for  symptom 
evidence  before  continuing  on  to  other  sections. 

Of  the  total  number  of  errors  committed  by  the  control 
group,  50%  were  classified  as  inappropriate  choice  of  procedure. 
Forty-five  per  cent  of  the  experimental  group  errors  were  so 
classified.  This  may  reflect  a  point  which  needs  to  be 
emphasized  in  the  current  program  of  instruction. 


3.2  Student  Acceptance 

The  student  opinionnaire  addressed  the  three  general 
categories  of  previous  computer  experience,  simulation 
instructions,  and  the  simulation  itself.  Of  the  27  students  who 
completed  the  opinionnaire,  all  but  one  had  had  at  least  some 
experience  with  video  games.  Thirty-seven  per  cent  of  the 
students  used  computers  in  high  school  and  an  equal  percentage 
used  computers  some  time  after  high  school. 
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Every  student  considered  the  verbal  instruction  presentation 
for  the  simulation  to  be  either  helpful  or  very  helpful. 
Eighty-one  per  cent  of  the  students  used  the  Student  Manual  for 
additional  instructions.  All  students  who  used  the  manual 
considered  it  to  be  either  helpful  or  very  helpful. 

Every  student  found  the  simulation  itself  interesting. 
Eighty-one  per  cent  of  the  students  believed  that  simulation 
practice  helped  them  in  troubleshooting  real  SB  3614  equipment. 
Further,  96%  believed  that  the  same  type  of  simulation  should  be 
used  in  other  36H  courses.  Concerning  the  issue  of  generalizable 
troubleshooting  skills.  89%  said  that  troubleshooting  skills 
learned  through  use  of  the  SB  3614  simulation  would  help  in 
troubleshooting  problems  in  other  courses  and  on  the  job. 
Overall,  all  of  the  students  were  satisfied  with  the  SB  3614 
computer-based  simulation  as  a  classroom  learning  aid.  The  final 
question  on  the  opinionnaire  asked  for  general  comments. 


3.3  Instructor  Acceptance 

After  the  simulation  had  been  used  by  the  experimental  group 
and  four  other  classes,  the  instructors  were  formally  asked  their 
opinions  of  the  simulation's  technical  content,  ease  of  use, 
instructional  effectiveness,  and  implications  for  the  Program  of 
Instruction.  In  general,  the  instructors  were  quite  positive 
about  the  simulation  and  all  seemed  to  accept  it  as  a  useful 
instructional  tool.  The  following  are  specific  comments  and 
suggestions  for  future  work  with  the  simulation. 

The  instructors  felt  that  the  simulation  provided  the  same 
types  of  information  that  would  be  available  using  real 
equipment.  The  FAULT  subsystem  diagrams  were  quite  helpful  for 
understanding  the  subsystems  of  the  switchboard.  In  fact,  some 
of  the  instructors  contemplated  using  the  simulation's  FAULT 
drawings  as  training  aids  for  teaching  logical  and  functional 
relationships  of  parts  in  the  various  subsystems. 

Concerning  ease  of  use  of  the  simulation,  it  was  noted  that 
students  could  learn  to  use  the  simulation  with  an  instructor's 
introduction  supplemented  with  consultation  of  the  Student 
Manual.  Instructors  found  that  their  introductions  were  most 
effective  when  they  included  a  demonstration  of  the  simulation. 
They  commented  that  the  FAULT  section  was  the  most  difficult 
portion  to  explain,  but  stated  that  the  FAULT  subsystem  diagrams 
were  helpful.  It  was  suggested  that  a  15  to  20  minute  video  tape 
be  made  for  introduction/demonstration  of  the  simulation. 

The  instructors  observed  that  students  enjoyed  the  challenge 
of  simulation  problems  but  were  frustrated  by  the  relatively  long 
computer  execution  time,  particularly  during  disk  access.  This 
problem  could  be  alleviated  with  the  use  of  a  more  efficient  and 
larger  storage  element  such  as  a  Winchester  hard  disk  drive  and 
with  increased  RAM. 
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The  instructors  were  reluctant  to  suggest  that  simulation 
practice  could  replace  real  equipment  practice.  They  indicated 
that  they  wanted  more  course  time  to  allow  them  to  increase  the 
complexity  of  the  real  equipment  problems. 

All  instructors  felt  that  the  simulation  should  become  a 
part  of  the  SB  3614  Program  of  Instruction.  They  also  commented 
that  the  ideal  number  of  hours  of  simulation  use  had  not  been 
determined  but  this  issue  can  be  resolved  in  the  future. 

In  summary,  the  instructors  were  very  supportive  of  the 
simulation  and  recommend  its  continued  use.  There  are  certain 
segments  of  the  simulation  that  could  be  modified  and  certainly 
the  computer  hardware  should  be  slightly  enhanced. 


3.4  Summary  of  Statistical  Results 

The  results  of  the  pilot  evaluation  suggest  that  live  system 
troubleshooting  performance  is  the  same  with  additional  training 
with  the  SB  3§14  simulation  as  it  is  with  additional  real 
equipment  training.  Based  on  the  performance  measures  of 
solution  success  and  time  to  solution,  the  experimental  group 
performed  as  well  as  the  control  group  on  the  real  equipment 
test.  Regarding  the  quality  of  the  troubleshooters’  methods, 
especially  with  respect  to  the  number  of  errors  made,  the 
experimental  group  had  better  performance. 

For  the  global  measures,  the  experimental  group  solved  one 
problem  out  of  three  in  significantly  less  time  than  the  control 
group.  There  was  no  difference  in  the  number  of  solution 
successes  between  the  groups. 

When  there  were  significant  differences  in  experts'  ratings 
of  individual  student  performance,  the  experimental  group  was 
rated  significantly  higher  than  the  control  group.  Two  experts 
gave  significantly  higher  composite  ratings  to  students  in  the 
experimental  group.  On  an  action-by-action  basis,  actions  of 
students  in  the  experimental  group  were  rated  significantly 
higher  than  those  of  the  control  group. 

A  significantly  higher  percentage  of  the  control  group's 
actions  were  considered  to  be  errors  by  the  experts.  More  data 
must  be  gathered  before  any  conclusions  concerning  errors  may  be 
drawn. 
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Student  response  to  the  simulation  as  recorded  via  the 
opinionnaire  was  very  positive.  Students,  both  with  and  without 
video  game  and  computer  experience,  found  the  simulation 
interesting.  They  also  believed  that  the  training  method  used  in 
the  simulation  would  help  them  troubleshoot  other  equipment  and 
that  the  same  type  of  simulation  should  be  used  in  other  courses. 
The  students  that  used  the  Student  Manual  found  it  to  be  helpful. 
Overall,  students  were  satisfied  with  the  simulation. 


4. 


CONCLUSIONS  AND  RECOMMENDATIONS 


This  section  provides  conclusions  and  recommendations  that 
are  specific  to  this  project,  and  others  that  are  meant  to  be 
generally  applicable  to  conceptualization,  design,  implementa¬ 
tion,  and  evaluation  of  computer-based  training  systems. 


4.1  Simulation  Strengths 

Generally,  the  project's  pilot  evaluation  showed  that  a 
microprocessor-based,  mixed-fidelity  simulation  can  be 
successfully  integrated  into  an  Army  electronics  course.  The 
computer  simulation  can  have  a  positive  effect  on  student 
troubleshooting  performance.  While  the  simulation  was  of  low  to 
moderate  fidelity  it  presented  challenging  troubleshooting 
practice  that  was  well  received  by  the  students  as  well  as  the 
instructors . 

The  results  from  a  pilot  test  cannot  be  considered  as 
conclusive  evidence.  However,  the  preliminary  data  suggest  that 
simulation-based  training  intermixed  with  real  equipment  practice 
may  be  more  desirable,  in  terms  of  post- training  troubleshooting 
performance,  than  use  of  real  equipment  exclusively.  Since  many 
aspects  of  real  equipment  troubleshooting  are  cognitive, 
involving  mental  processing  of  displayed  symptomatic  information, 
there  is  not  a  great  need  for  high-fidelity  hardware.  In  fact, 
computer-based,  mixed-fidelity  simulation  has  the  potential  to 
present  considerably  more  troubleshooting  practice  than  does  real 
equipment.  In  addition,  the  computer  can  keep  records  of  student 
performance,  provide  appropriate  feedback  regarding  errors,  and 
permit  a  student  to  work  independently  at  a  personalized  pace. 
The  computer  simulation  can  also  provide  a  framework  for 
generalized  troubleshooting  while  delivering  equipment  specific 
diagnostic  experience. 

Another  very  important  aspect  of  simulation-based  diagnostic 
training  is  its  interactive  nature.  The  simulation  permits  the 
learner  to  actively  engage  in  problem  solving  without  the  fear  of 
personal  injury  or  damage  to  the  expensive  real  equipment.  This 
heavy  emphasis  on  learner  involvement  is  of  significant 
instructional  value  and  cannot  be  overstated. 


4.2  Simulation  Weaknesses 

The  pilot  evaluation  exposed  a  few  problems  that  can  be 
corrected.  The  shortcomings  of  the  SB  3614  simulation  were 
primarily  attributable  to  microprocessor  hardware  constraints  and 
did  not  seem  to  have  a  significant  effect  on  the  simulation's 
training  effectiveness.  The  nature  of  the  simulation  design 
resulted  in  a  situation  where  the  computer  hardware  was  taxed  to 
its  limit  in  terms  of  RAM  and  disc  storage.  As  previously 
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mentioned,  approximately  98%  of  system  capacity  was  used.  The 
result,  as  expected,  was  relatively  slow  loading  and  execution 
time  for  certain  segments  of  the  simulation.  Since  the  entire 
program  was  too  large  for  RAM,  program  segments  were  loaded  only 
when  requested.  Increasing  RAM  would  have  alleviated  the  problem 
slightly,  however  access  time  would  have  remained  high  due  to  the 
nature  of  the  8-bit  machine  and  the  use  of  the  high-level 
programming  language  rather  than  assembly  language.  Other 
shortcomings  related  to  memory,  storage,  and  access  were  the 
inability  to  select  and  use  special  purpose  keys  (like  "help") 
and  long  delays  while  information  for  each  new  screen  was 
retrieved  from  the  disc.  Both  of  these  problems  stemmed  from 
insufficient  memory. 

The  resolution  and  quality  of  the  color  graphics  was  another 
significant  shortcoming.  Color  graphics  would  have  made  the 
simulation  more  attractive  to  the  student  user.  While  color 
graphics  may  not  have  affected  the  information  transfer  from  the 
display,  it  may  have  enhanced  the  student's  enthusiasm  towards 
using  the  simulation.  The  poor  quality  of  text  written  on  the 
graphics  screen  prohibited  the  use  of  color  graphics  for  this 
project. 

Another  problem  related  to  the  use  of  microprocessor 
hardware  was  data  collection.  The  limited  storage  capacity  of 
the  standard  diskette  made  the  logistics  of  data  collection 
rather  tedious.  While  these  problems  could  be  alleviated  with  a 
hard  disc,  the  systems  used  for  this  project  were  not  so 
equipped. 

There  are  a  number  of  solutions  to  the  hardware  deficiencies 
which  can  be  briefly  mentioned.  The  final  choice,  of  course, 
must  be  defined  by  such  requirements  as  the  necessary  level  of 
fidelity,  the  need  for  extensive  data  collection,  the 
availability  of  hardware  for  student  stations,  and  the  resources 
available  for  courseware  development. 

The  hardware  solutions  fall  into  three  basic  categories. 
These  solution  categories  are  the  following:  expand  the  present 
Apple  system;  use  a  16  or  32  bit  computer  and  have  multiple 
users  for  each  computer;  or  use  videodisc  technology. 

The  first  solution,  Apple  upgrade,  would  require  the 
necessary  peripherals  to  increase  RAM  and  disc  storage,  and 
decrease  disc  access  time.  A  high  resolution  color  terminal  and 
improved  Apple  graphics  capabilities  would  also  be  needed. 
Another  solution  in  this  category  would  be  to  convert  from  Pascal 
to  ASSEMBLER. 

The  second  solution  hinges  on  increased  computer  power.  A 
larger  computer  (i.e.,  16  or  32  bit)  would  be  much  faster  in 
execution  time  and  more  compatible  to  increased  RAM  and  hard  disc 
storage.  This  system  would  require  a  high  resolution  color 
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graphics  terminal  for  each  student  station.  A  mini  or  super 
micro,  would  likely  support  from  three  to  six  student  stations  as 
the  simulation  was  configured  for  this  project. 

*  The  third  solution  is  videodisc.  This  technology  would 
require  the  videodisc  player  and  interface  to  the  micro  computer. 
If  videodisc  was  used  it  would  permit  higher  fidelity 
representation  of  the  switchboard  than  was  possible  with  computer 
graphics.  Further,  the  videodisc  could  store  all  of  the  pictures 
that  were  generated  by  the  program.  This  videodisc  picture 
storage  would  reduce  the  demands  on  RAM  and  diskette  storage  and 
thus  make  the  personal  micro  a  quite  feasible  alternative  again. 

Whatever  the  hardware  choice,  the  overall  effectiveness  of  a 
training  device  is  most  dependent  on  the  pedagogical 
considerations  and  organized  systematic  design.  The  training 
design  decisions  must  ultimately  drive  the  hardware  choice. 


4.3  Cost  Effectiveness  Implications 

The  extent  to  which  the  use  of  computer-based  training  can 
be  optimally  mixed  with  real  equipment  practice  certainly  needs 
further  study.  The  authors  contend  that  the  mix  of  real 
equipment  and  computer  simulation  training  provides  excellent 
instruction  for  troubleshooting.  By  exchanging  inexpensive 
microprocessors  for  a  portion  of  the  real  equipment  units  there 
could  be  a  reduction  in  training  costs.  Cost  reduction  is 
desirable  only  if  the  quality  of  the  training  is  not  adversely 
affected.  In  the  case  of  this  computer  simulation,  it  may  be 
successfully  argued  that  simulation  combined  with  real  equipment 
practice  for  a  given  time  period  is  better  than  real  equipment 
practice  alone  for  the  same  time  duration.  The  formal  evaluation 
of  training  and  retention  effectiveness  planned  for  the  second 
year  of  this  project  will  certainly  provide  a  thorough  test  of 
this  hypothesis. 

While  hardware  procurement  is  one  cost  of  training,  the 
higher  costs  are  often  student  and  instructor  salaries.  The 
computer  will  not  replace  the  invaluable  instructor.  However, 
computer  stations  have  the  potential  to  increase  instructor 
efficiency.  In  a  normal,  real-equipment  troubleshooting 
laboratory,  one  very  busy  instructor  may  be  able  to  supervise  ten 
students.  With  the  ten  to  one  ratio,  students  may  find 
themselves  waiting  to  receive  their  portion  of  instructor 
attention.  It  is  reasonable  to  expect  that  class  size  could  be 
expanded  as  the  number  of  computer-based  work  stations  is 
expanded.  Of  course,  the  number  of  added  work  stations  would 
depend  on  a  variety  of  economic  and  organizational  tradeoffs,  the 
resolution  of  which  was  not  part  of  the  work  reported  here. 
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4.4  Summar 


The  SB  3614  software/courseware  has  been  developed  and 
implemented  at  the  Signal  Center.  Preliminary  analysis  indicated 
that  the  additional  computer-based  experimental  training  resulted 
in  post- training  troubleshooting  performance  comparable  to  that 
achieved  through  additional  real  equipment  practice.  The 
simulation  is  acceptable  to  the  instructors  and  to  the  student 
users.  On-going  evaluation  and  modification  to  provide 
continuing  support  to  the  SB  3614  simulation  should  be 
established.  Such  support  will  guarantee  the  continued  use  of 
this  courseware  and  open  the  way  for  additional  developments  in 
areas  of  need. 
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